PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

IPO currently divides the shared per-token log-probabilities by loss_mask.sum(-1) before reducing them into sequences. This can fail broadcasting for padded batches, uses the entire packed row as the denominator for padding-free/packed batches, and also normalizes the sigmoid component when sigmoid and IPO are combined.

Keep forward outputs and cached reference log-probabilities as sequence sums. Count valid completion tokens per sequence using the existing vectorized packed reduction, then normalize both policy and reference scores only for the IPO component. Clamp counts to at least one for empty completions.

Preserve logps/chosen and logps/rejected as sums and report logps_mean/chosen and logps_mean/rejected for IPO. These metrics use the existing trainer logging path; with --report_to wandb, the W&B callback logs train/logps_mean/{chosen,rejected} and eval/logps_mean/{chosen,rejected}.

Only swift/rlhf_trainers/dpo_trainer.py changes: 29 insertions and 9 deletions.

Experiment results

CPU validation with PyTorch 2.13.0, Transformers 5.14.1 and TRL 1.9.2:

• 48 checks passed for padded, padding-free, packing, the sequence-parallel post-gather contract, online/cached reference scores, standalone and mixed losses in both orders, gradients, empty completions, existing length-dependent weighting, and metrics.
• Two additional train/eval checks passed through the actual store_metrics -> Trainer.log -> WandbCallback path with a captured W&B API call. No live W&B run was created.
• The same checks against main reproduce the padded broadcasting error and incorrect standalone/mixed IPO loss values; standalone sigmoid matches the independent formula.
• uvx pre-commit run --all-files and git diff --check passed.

Validation used tiny CPU logits; a full multi-GPU training run was not performed.